{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Relation Extraction\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deeppavlov/DeepPavlov/blob/master/docs/features/models/relation_extraction.ipynb)\n",
    "\n",
    "# Table of contents \n",
    "\n",
    "1. [Introduction to the task](#1.-Introduction-to-the-task)\n",
    "\n",
    "2. [Get started with the model](#2.-Get-started-with-the-model)\n",
    "\n",
    "3. [Models list](#3.-Models-list)\n",
    "\n",
    "4. [Use the model for prediction](#4.-Use-the-model-for-prediction)\n",
    "\n",
    "    4.1 [Predict using Python](#4.1-Predict-using-Python)\n",
    "    \n",
    "    4.2 [Predict using CLI](#4.2-Predict-using-CLI)\n",
    "\n",
    "5. [Customize the model](#5.-Customize-the-model)\n",
    "    \n",
    "    5.1 [Description of config parameters](#5.1-Description-of-config-parameters)\n",
    "    \n",
    "    5.2 [Train Relation Extraction on custom data](#5.2-Train-Relation-Extraction-on-custom-data)\n",
    "\n",
    "6. [Relations list](#6.-Relations-list)\n",
    "\n",
    "    6.1 [Relations used in English model](#6.1-Relations-used-in-English-model)\n",
    "    \n",
    "    6.2 [Relations used in Russian model](#6.2-Relations-used-in-Russian-model)\n",
    "\n",
    "# 1. Introduction to the task\n",
    "\n",
    "Relation extraction is the task of detecting and classifying the relationship between two entities in text.\n",
    "DeepPavlov provides the document-level relation extraction meaning that the relation can be detected between the entities that are not in one sentence.\n",
    "\n",
    "**RE Model Architecture**\n",
    "\n",
    "We based our model on the [Adaptive Thresholding and Localized Context Pooling](https://arxiv.org/pdf/2010.11304.pdf) model and used NER entity tags as additional input. Two core ideas of this model are:\n",
    "\n",
    "- Adaptive Threshold\n",
    "\n",
    "The usual global threshold for converting the RE classifier output probability to relation label is replaced with a learnable one. A new threshold class that learns an entities-dependent threshold value is introduced and learnt as all other classes. During prediction the positive classes (= relations that are hold in the sample indeed) are claimed to be the classes with higher logins that the TH class, while all others are negative ones.\n",
    "\n",
    "- Localised Context Pooling\n",
    "\n",
    "The embedding of each entity pair is enhanced with an additional local context embedding related to both entities. Such representation, which is attended to the relevant context in the document, is useful to decide the relation for exactly this entity pair. For incorporating the context information the attention heads are directly used.\n",
    "\n",
    "# 2. Get started with the model\n",
    "\n",
    "First make sure you have the DeepPavlov Library installed.\n",
    "[More info about the first installation.](http://docs.deeppavlov.ai/en/master/intro/installation.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q deeppavlov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before using the model make sure that all required packages are installed running the command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov install re_docred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Models list\n",
    "\n",
    "The table presents a list of all of the relation extraction models available in the DeepPavlov Library.\n",
    "\n",
    "| Config | Language | Dataset |\n",
    "| :--- | :---: | :--- |\n",
    "| relation_extraction/re_docred.json | En | [DocRED](https://www.aclweb.org/anthology/P19-1074/) |\n",
    "| relation_extraction/re_rured.json | Ru | [RuRED](http://www.dialog-21.ru/media/5093/gordeevdiplusetal-031.pdf) |\n",
    "\n",
    "## Some details on DocRED corpus English RE model was trained on\n",
    "\n",
    "The English RE model was trained on DocRED English corpus. It was constructed from Wikipedia and Wikidata and is now the largest human-annotated dataset for document-level RE from plain text.\n",
    "\n",
    "As the original DocRED test dataset containes only unlabeled data, while we want to have labeled one in order to perform evaluation, we decided to:\n",
    "1. merge train and dev data (= labeled data)\n",
    "2. split them into new train, dev and test dataset\n",
    "\n",
    "Currently, there are two types of possible splittings provided:\n",
    "\n",
    "- user can set the relative size of dev and test data (e.g. 1/7)\n",
    "- user can set the absolute size of dev and test data (e.g. 2000 samples)\n",
    "\n",
    "In our experiment, we set the absolute size of dev and test data == 150 initial documents. It resulted in approximately 3500 samples.\n",
    "\n",
    "We additionally generate negative samples if it was necessary to have the following proportions:\n",
    "- for train set: negative samples are twice as many as positive ones\n",
    "- for dev & test set: negative samples are the same amount as positive ones\n",
    "\n",
    "| Train | Dev | Test |\n",
    "| :---: | :---: | :---: |\n",
    "| 130650 | 3406 | 3545 |\n",
    "\n",
    "| Train Positive | Train Negative | Dev Positive   | Dev Negative   | Test Positive  | Test Negative  |\n",
    "| :---: | :---: | :---: | :---: | :---: | :---: |\n",
    "| 44823          | 89214          | 1239           | 1229           | 1043           | 1036           |\n",
    "\n",
    "## Some details on RuRED corpus Russian RE model was trained on\n",
    "\n",
    "In case of RuRED we used the train, dev and test sets from the original RuRED setting. We additionally generate negative samples if it was necessary to have the following proportions:\n",
    "\n",
    "- for train set: negative samples are twice as many as positive ones\n",
    "- for dev & test set: negative samples are the same amount as positive ones\n",
    "\n",
    "| Train         | Dev           | Test           |\n",
    "| :---: | :---: | :---: |\n",
    "| 12855         | 1076          |1072            |\n",
    "\n",
    "| Train Positive | Train Negative | Dev Positive | Dev Negative | Test Positive | Test Negative |\n",
    "| :---: | :---: | :---: | :---: | :---: | :---: |\n",
    "| 4285           | 8570           | 538          | 538          | 536           | 536           |\n",
    "\n",
    "# 4. Use the model for prediction\n",
    "\n",
    "## 4.1 Predict using Python\n",
    "\n",
    "### English"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import configs, build_model\n",
    "\n",
    "re_model = build_model(configs.relation_extraction.re_docred, download=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['P26'], ['spouse']]\n"
     ]
    }
   ],
   "source": [
    "sentence_tokens = [[\"Barack\", \"Obama\", \"is\", \"married\", \"to\", \"Michelle\", \"Obama\", \",\", \"born\", \"Michelle\", \"Robinson\", \".\"]]\n",
    "entity_pos = [[[(0, 2)], [(5, 7), (9, 11)]]]\n",
    "entity_tags = [[\"PER\", \"PER\"]]\n",
    "pred = re_model(sentence_tokens, entity_pos, entity_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Model Input**:\n",
    "\n",
    "- list of tokens of a text document\n",
    "- list of entities positions (i.e. all start and end positions of both entities' mentions)\n",
    "- list of NER tags of both entities.\n",
    "\n",
    "As NER tags, we adapted the used in the DocRED corpus, which are, in turn, inherited from [Tjong Kim Sang and De Meulder(2003)](https://aclanthology.org/W03-0419/)\n",
    "\n",
    "**The whole list of 6 English NER tags**\n",
    "\n",
    "| Tag | Description |\n",
    "| :--- | :--- |\n",
    "|PER | People, including fictional |\n",
    "|ORG    | Companies, universities, institutions, political or religious groups, etc.                     |\n",
    "|LOC    | Geographically defined locations, including mountains, waters, etc. <br> Politically defined locations, including countries, cities, states, streets, etc. <br> Facilities, including buildings, museums, stadiums, hospitals, factories, airports, etc.       |\n",
    "|TIME   | Absolute or relative dates or periods.                                                         |\n",
    "|NUM    | Percents, money, quantities                                                                    |\n",
    "|MISC   | Products, including vehicles, weapons, etc. <br> Events, including elections, battles, sporting MISC events, etc. Laws, cases, languages, etc.   |\n",
    "\n",
    "**Model Output**: one or several of the [97 relations](#6.1-Relations-used-in-English-model) found between the given entities; relation id in [Wikidata](https://www.wikidata.org/wiki/Wikidata:Main_Page) (e.g. 'P26') and relation name ('spouse').\n",
    "\n",
    "### Russian"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import configs, build_model\n",
    "\n",
    "re_model = build_model(configs.relation_extraction.re_rured)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['P495'], ['страна происхождения']]\n"
     ]
    }
   ],
   "source": [
    "sentence_tokens = [[\"Илон\", \"Маск\", \"живет\", \"в\", \"Сиэттле\", \".\"]]\n",
    "entity_pos = [[[(0, 2)], [(4, 5)]]]\n",
    "entity_tags = [[\"PERSON\", \"CITY\"]]\n",
    "pred = re_model(sentence_tokens, entity_pos, entity_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Model Input**:\n",
    "\n",
    "- list of tokens of a text document\n",
    "- list of entities positions (i.e. all start and end positions of both entities' mentions)\n",
    "- list of NER tags of both entities.\n",
    "\n",
    "**Model Output**: one or several of the [30 relations](#6.2-Relations-used-in-Russian-model) found between the given entities; a Russian relation name (e.g. \"участник\") or an English one, if Russian one is unavailable, and, if applicable, its id in [Wikidata](https://www.wikidata.org/wiki/Wikidata:Main_Page) (e.g. 'P710').\n",
    "\n",
    "## 4.2 Predict using CLI\n",
    "\n",
    "You can also get predictions in an interactive mode through CLI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m deeppavlov interact re_docred [-d]\n",
    "! python -m deeppavlov interact re_rured [-d]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`-d` is an optional download key (alternative to `download=True` in Python code). It is used to download the pre-trained model along with embeddings and all other files needed to run the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Customize the model\n",
    "\n",
    "## 5.1 Description of config parameters\n",
    "\n",
    "Parameters of ``re_preprocessor`` component:\n",
    "\n",
    "- ``ner_tags: List[str]`` - ner tags of the entities, which are one-hot encoded and concatenated to entity embeddings in the output of the Transformer;\n",
    "- ``special_token: str`` - the token which is added before and after the entities (subject and object in the triplet) mentions;\n",
    "- ``default_tag: str`` - default ner tags, if no tags are provided;\n",
    "- ``do_lower_case: bool`` - set True if lowercasing is needed.\n",
    "\n",
    "Parameters of ``re_classifier`` component:\n",
    "\n",
    "- ``n_classes: int`` - number of relations which the model supports;\n",
    "- ``num_ner_tags: int`` - number of ner tags;\n",
    "- ``return_probas: bool`` - whether to return confidences of predicted relations.\n",
    "\n",
    "Parameters of ``re_postprocessor`` component:\n",
    "    \n",
    "- ``rel2id_path: str`` - the file with mapping of relation IDs in the knowledge base to relation number (for example, \"P19\": 24);\n",
    "- ``rel2label_path: str`` - the file with mapping of relation IDs to relation labels.\n",
    "\n",
    "## 5.2 Train Relation Extraction on custom data\n",
    "\n",
    "There are two kinds of dataset readers for relation extraction in DeepPavlov library:\n",
    "\n",
    "- ``docred_reader``, which takes into account partition of the text into sentences and several mentions in the text for one entity;\n",
    "- ``rured_reader``, a simplified dataset reader.\n",
    "\n",
    "### Train with ``docred_reader``\n",
    "\n",
    "You should prepare **train_annotated.json**, **dev.json**, **test.json** in the following format:\n",
    "\n",
    "    {\n",
    "      \"vertexSet\": [\n",
    "        [\n",
    "          {\n",
    "            \"name\": entity1_mention1,\n",
    "            \"pos\": [mention1 start token index, mention1 end token index],\n",
    "            \"sent_id\": ID of the sentence with the entity1 mention1,\n",
    "            \"type\": ner tag\n",
    "          },\n",
    "          {\n",
    "            \"name\": entity1_mention2,\n",
    "            ...\n",
    "          },\n",
    "          ...\n",
    "        ],\n",
    "        [ ... ]\n",
    "      ],\n",
    "      \"labels\": [\n",
    "        {\n",
    "          \"r\": relation ID,\n",
    "          \"h\": index of head entity of the triplet in the vertexSet list,\n",
    "          \"t\": index of tail entity of the triplet in the vertexSet list,\n",
    "          \"evidence\": [\n",
    "            indices of the sentences with the triplet\n",
    "          ]\n",
    "        },\n",
    "        ...\n",
    "      ],\n",
    "      \"title\": doc title,\n",
    "      \"sentences\": [\n",
    "        list of tokens of sentence 1,\n",
    "        list of tokens of sentence 2,\n",
    "        ...\n",
    "      ],\n",
    "      ...\n",
    "    }\n",
    "\n",
    "For example,\n",
    "\n",
    "    {\n",
    "      \"vertexSet\": [\n",
    "        [\n",
    "          {\n",
    "            \"name\": \"Elon Musk\",\n",
    "            \"pos\": [0, 2],\n",
    "            \"sent_id\": 0,\n",
    "            \"type\": \"PER\"\n",
    "          }\n",
    "        ],\n",
    "        [\n",
    "          {\n",
    "            \"name\": \"Seattle\",\n",
    "            \"pos\": [4, 5],\n",
    "            \"sent_id\": 0,\n",
    "            \"type\": \"CITY\"\n",
    "          }\n",
    "        ]\n",
    "      ],\n",
    "      \"labels\": [\n",
    "        {\n",
    "          \"r\": \"P551\",\n",
    "          \"h\": 0,\n",
    "          \"t\": 1,\n",
    "          \"evidence\": [0]\n",
    "        }\n",
    "      ],\n",
    "      \"title\": \"title1\",\n",
    "      \"sentences\": [\n",
    "        [\"Elon\", \"Musk\", \"lives\", \"in\", \"Seattle\", \".\"]\n",
    "      ]\n",
    "    }\n",
    "\n",
    "### Train with  ``rured_reader``\n",
    "\n",
    "You should prepare **train.json**, **dev.json**, **test.json** in the following format:\n",
    "\n",
    "    {\n",
    "        \"token\": list of text tokens,\n",
    "        \"relation\": relation ID,\n",
    "        \"subj_start\": index of the token of the subject start in the list,\n",
    "        \"subj_end\": index of the token of the subject end in the list,\n",
    "        \"obj_start\": index of the token of the object start in the list,\n",
    "        \"obj_end\": index of the token of the object end in the list,\n",
    "        \"subj_type\": ner tag of the subject entity,\n",
    "        \"obj_type\": ner tag of the object entity,\n",
    "    },\n",
    "\n",
    "for example:\n",
    "\n",
    "    {\n",
    "        \"token\": [\"Илон\", \"Маск\", \"живет\", \"в\", \"Сиэттле\", \".\"],\n",
    "        \"relation\": \"P551\",\n",
    "        \"subj_start\": 0,\n",
    "        \"subj_end\": 2,\n",
    "        \"obj_start\": 4,\n",
    "        \"obj_end\": 5,\n",
    "        \"subj_type\": \"PERSON\",\n",
    "        \"obj_type\": \"CITY\"\n",
    "    }\n",
    "\n",
    "#### Train the model using Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import train_model\n",
    "\n",
    "train_model(\"re_docred\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**or using CLI:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m deeppavlov train re_docred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Relations list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 Relations used in English model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Relation id     |  Relation                                           |\n",
    "| :--- | :--- |\n",
    "|P6              |  head of government                                 |\n",
    "|P17             |  country                                            |\n",
    "|P19             |  place of birth                                     |\n",
    "|P20             |  place of death                                     |\n",
    "|P22             |  father                                             |\n",
    "|P25             |  mother                                             |\n",
    "|P26             |  spouse                                             |\n",
    "|P27             |  country of citizenship                             |\n",
    "|P30             |  continent                                          |\n",
    "|P31             |  instance of                                        |\n",
    "|P35             |  head of state                                      |\n",
    "|P36             |  capital                                            |\n",
    "|P37             |  official language                                  |\n",
    "|P39             |  position held                                      |\n",
    "|P40             |  child                                              |\n",
    "|P50             |  author                                             |\n",
    "|P54             |  member of sports team                              |\n",
    "|P57             |  director                                           |\n",
    "|P58             |  screenwriter                                       |\n",
    "|P69             |  educated at                                        |\n",
    "|P86             |  composer                                           |\n",
    "|P102            |  member of political party                          |\n",
    "|P108            |  employer                                           |\n",
    "|P112            |  founded by                                         |\n",
    "|P118            |  league                                             |\n",
    "|P123            |  publisher                                          |\n",
    "|P127            |  owned by                                           |\n",
    "|P131            |  located in the administrative territorial entity   |\n",
    "|P136            |  genre                                              |\n",
    "|P137            |  operator                                           |\n",
    "|P140            |  religion                                           |\n",
    "|P150            |  contains administrative territorial entity         |\n",
    "|P155            |  follows                                            |\n",
    "|P156            |  followed by                                        |\n",
    "|P159            |  headquarters location                              |\n",
    "|P161            |  cast member                                        |\n",
    "|P162            |  producer                                           |\n",
    "|P166            |  award received                                     |\n",
    "|P170            |  creator                                            |\n",
    "|P171            |  parent taxon                                       |\n",
    "|P172            |  ethnic group                                       |\n",
    "|P175            |  performer                                          |\n",
    "|P176            |  manufacturer                                       |\n",
    "|P178            |  developer                                          |\n",
    "|P179            |  series                                             |\n",
    "|P190            |  sister city                                        |\n",
    "|P194            |  legislative body                                   |\n",
    "|P205            |  basin country                                      |\n",
    "|P206            |  located in or next to body of water                |\n",
    "|P241            |  military branch                                    |\n",
    "|P264            |  record label                                       |\n",
    "|P272            |  production company                                 |\n",
    "|P276            |  location                                           |\n",
    "|P279            |  subclass of                                        |\n",
    "|P355            |  subsidiary                                         |\n",
    "|P361            |  part of                                            |\n",
    "|P364            |  original language of work                          |\n",
    "|P400            |  platform                                           |\n",
    "|P403            |  mouth of the watercourse                           |\n",
    "|P449            |  original network                                   |\n",
    "|P463            |  member of                                          |\n",
    "|P488            |  chairperson                                        |\n",
    "|P495            |  country of origin                                  |\n",
    "|P527            |  has part                                           |\n",
    "|P551            |  residence                                          |\n",
    "|P569            |  date of birth                                      |\n",
    "|P570            |  date of death                                      |\n",
    "|P571            |  inception                                          |\n",
    "|P576            |  dissolved, abolished or demolished                 |\n",
    "|P577            |  publication date                                   |\n",
    "|P580            |  start time                                         |\n",
    "|P582            |  end time                                           |\n",
    "|P585            |  point in time                                      |\n",
    "|P607            |  conflict                                           |\n",
    "|P674            |  characters                                         |\n",
    "|P676            |  lyrics by                                          |\n",
    "|P706            |  located on terrain feature                         |\n",
    "|P710            |  participant                                        |\n",
    "|P737            |  influenced by                                      |\n",
    "|P740            |  location of formation                              |\n",
    "|P749            |  parent organization                                |\n",
    "|P800            |  notable work                                       |\n",
    "|P807            |  separated from                                     |\n",
    "|P840            |  narrative location                                 |\n",
    "|P937            |  work location                                      |\n",
    "|P1001           |  applies to jurisdiction                            |\n",
    "|P1056           |  product or material produced                       |\n",
    "|P1198           |  unemployment rate                                  |\n",
    "|P1336           |  territory claimed by                               |\n",
    "|P1344           |  participant of                                     |\n",
    "|P1365           |  replaces                                           |\n",
    "|P1366           |  replaced by                                        |\n",
    "|P1376           |  capital of                                         |\n",
    "|P1412           |  languages spoken, written or signed                |\n",
    "|P1441           |  present in work                                    |\n",
    "|P3373           |  sibling                                            |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 Relations used in Russian model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Relation                   | Relation id       | Russian relation                |\n",
    "| :--- | :--- | :--- |\n",
    "| MEMBER                     | P710              | участник                        |\n",
    "| WORKS_AS                   | P106              | род занятий                     |\n",
    "| WORKPLACE                  | --                | --                              |\n",
    "| OWNERSHIP                  | P1830             | владеет                         |\n",
    "| SUBORDINATE_OF             | --                | --                              |\n",
    "| TAKES_PLACE_IN             | P276              | местонахождение                 |\n",
    "| EVENT_TAKES_PART_IN        | P1344             | участвовал в                    |\n",
    "| SELLS_TO                   | --                | --                              |\n",
    "| ALTERNATIVE_NAME           | --                | --                              |\n",
    "| HEADQUARTERED_IN           | P159              | расположение штаб-квартиры      |\n",
    "| PRODUCES                   | P1056             | продукция                       |\n",
    "| ABBREVIATION               | --                | --                              |\n",
    "| DATE_DEFUNCT_IN            | P576              | дата прекращения существования  |\n",
    "| SUBEVENT_OF                | P361              | часть от                        |\n",
    "| DATE_FOUNDED_IN            | P571              | дата основания/создания/возн-я  |\n",
    "| DATE_TAKES_PLACE_ON        | P585              | момент времени                  |\n",
    "| NUMBER_OF_EMPLOYEES_FIRED  | --                | --                              |\n",
    "| ORIGINS_FROM               | P495              | страна происхождения            |\n",
    "| ACQUINTANCE_OF             | --                | --                              |\n",
    "| PARENT_OF                  | P40               | дети                            |\n",
    "| ORGANIZES                  | P664              | организатор                     |\n",
    "| FOUNDED_BY                 | P112              | основатель                      |\n",
    "| PLACE_RESIDES_IN           | P551              | место жительства                |\n",
    "| BORN_IN                    | P19               | место рождения                  |\n",
    "| AGE_IS                     | --                | --                              |\n",
    "| RELATIVE                   | --                | --                              |\n",
    "| NUMBER_OF_EMPLOYEES        | P1128             | число сотрудников               |\n",
    "| SIBLING                    | P3373             | брат/сестра                     |\n",
    "| DATE_OF_BIRTH              | P569              | дата рождения                   |"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 4
}
